Scheduling Data Intensive Workflow Applications based on Multi-Source Parallel Data Retrieval in Distributed Computing Networks

Many large-scale scientific experiments are carried out in collaboration with researchers and laboratories located around the world so that they can leverage expertise and high-tech infrastructures present at those locations and collectively perform experiments quicker. Data produced by these experiments are thus replicated and gets cached at multiple geographic locations. This necessitates new techniques for selection of both data and compute resources so that executions of applications are time and cost efficient when using distributed resources. Existing heuristics based techniques select ‘best’ data source for retrieving data to a compute resource and then carry out task-resource assignment. But, this approach of scheduling, which is based only on single source data retrieval, may not give time (and cost) efficient schedules when: 1) tasks are interdependent on data (workflow), 2) average size of data processed by every task is large, and 3) data transfer time exceeds task computation time by at least an order of magnitude. To achieve time efficient schedules, we leverage the presence of replicated data sources to retrieve data in parallel from multiple sources by incorporating the technique in our scheduling heuristic. In this paper, we proposed multi-source data retrieval based scheduling heuristic that assign interdependent tasks to compute resources based on both multi-source parallel data retrieval time and task-computation time. We carried out scheduling experiments by modeling applications from life sciences and astronomy domains and deploying them on both emulated and real testbed environments. Hence, with a combination of data retrieval and task-resource mapping technique, we showed that our heuristic can achieve time-efficient schedules that are better than existing heuristic based techniques, for scheduling application workflows.